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Abstract. For a smoothly bounded pseudoconvex domain D C C" of finite type 
with non-compact holomorphic automorphism group Aut(Z)), we show that the set 
S{D) of all boundary accumulation points for Aut(Z?) is a compact subset of dD 
and, if S{D) contains at least three points, it is a perfect set and thus has the power 
of the continuum. Moreover, we show that in this case, S{D) is either connected or 
the number of its connected components is uncountable. We also discuss how S{D) 
relates to other invariant subsets of dD. 



Suppose that D C C"^ is a smoothly bounded domain, i.e. D is bounded and dD 
is C°°-smooth. We assume that the group Aut(D) of holomorphic automorphisms 
of D is non-compact; this means, thanks to a classical result of H. Cartan, that 
there is a point q G dD such that for some p E D and a sequence {fj} C Aut(L>), 
one has that fj{p) — > g as j ^ cxd. Such a point q is called a boundary accumulation 
point for Aut{D) (see [GKl] for a discussion of this matter). 

Let S{D) denote the set of all boundary accumulation points for Aut(D). Exist- 
ing examples of domains with non-compact automorphism groups (see [FIK] for a 
discussion of the case of Reinhardt domains), for which the set S{D) can be found 
explicitly, indicate that this set should enjoy some explicit regularity properties. 

For instance, let us for the moment restrict attention to smoothly bounded Rein- 
hardt domains. It follows from [FIK] that, for such a domain D, S{D) is always a 
compact, connected smooth submanifold of dD. For such domains one can also ob- 
serve other interesting properties of S{D) such as the constancy and minimality of 
the rank of the Levi form of dD along S{D) (see [H] for the genesis of these ideas); 
there is also a certain relation between this rank and the dimensions of the orbits of 
the action of Aut(il>) on D. Similarly, the type in the sense of D'Angelo [D'Al] is 
constant and maximal along S{D). Many of these properties, when considered for 
general domains, appear to be related to the conjecture of Greene/Krantz [GK2] 
which states that every boundary accumulation point for a smoothly bounded do- 
main must be of finite type. 

In the present paper we begin a systematic study of the set S{D) for a fairly 
general class of domains, and obtain foundational results on its topology and the 
relation of S{D) to other invariant subsets of dD. We thank H. Boas, R. Remmert, 
J. Wolf and S. Fu for stimulating remarks and suggestions concerning this work. 
We are also grateful to K. Diederich for a very valuable discussion of the results of 
this paper. 
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We say that dD is variety-free at g e dD if there are no non-trivial germs of 
complex varieties lying in dD and passing through q. 

Proposition 1. Let D he a hounded domain in C". Suppose that dD is variety-free 
at each point of S{D). Then S{D) is compact. 

Proof. We need only to prove that S{D) is closed. Let {qk} be a sequence of points 
from S{D) such that q^ ^ q E dD as A; — > oo. Since dD is variety-free at each point 
qk, then for every qk there is a sequence {/^} from Aut(-D) such that converges 
to the constant map qk in all of D as j — oo (see [GKl]). 

Fix now a sequence {e^}, > 0, e/j — > as A; ^ oo. Next, fix a point p E D and 

for every k find an index j{k) such that \fl^'^\p) — < e^. It is now obvious that 

fk'^\p) — >■ 5 as /c — > cxD. Hence q G S{D) and S{D) is closed. 

The proposition is proved. ■ 



Remark. For smoothly bounded domains the variety-free assumption in Proposi- 
tion 1 would follow from the conjecture of Greene/Krantz. 

Theorem 2. Suppose that D C C"^ is a smoothly hounded pseudoconvex domain of 
finite type. Then, if S{D) contains at least three points, it is a perfect set and thus 
has the power of the continuum. Moreover, in this case, S{D) is either connected, 
or the number of its connected components is uncountable. 

Proof. Wc note that any automorphism of D extends to a C°°-automorphism of D 
(see e.g. [D'A2]). 

Assume that S{D) contains at least three points. We will first show that S{D) 
cannot have isolated points. Indeed, let q e S{D) be an isolated point. Let {fj} 
be a sequence in Aut{D) such that fj q m all of _D as j — >^ oo. By passing to a 
subsequence we can also assume that f~^ — > r in all of D where r G S{D). 

Suppose first that r = q. Since S{D) contains at least three points, one can find 
two distinct points s,t E S{D), s,t ^ q. Then by Theorem 1 of [B], fj{s) — * q, 
fj{t) — > as J — > oo. Since q is an isolated point of S{D) and each fj preserves 
S{D), we conclude that, for all sufiiciently large j, one has fj{s) = fj{t) = q. This 
is impossible since every fj is a one-to-one mapping on D. 

Suppose now that r ^ q. Then there is s G S{D) such that s ^ q,r. Then by 
[B], fj{q) q, fj{s) q as j ^ oo which implies as before that for all sufficiently 
large j, fj{q) — fj{s) = q, but this is again impossible since the fj are one-to-one 
on D. Thus if S{D) has at least three elements then S{D) does not have isolated 
points, and thus is a perfect set. 

Assume now that S{D) is disconnected and the number of its connected com- 
ponents is not uncountable. Let S{D) — [JkSk{D) be the decomposition of S{D) 
into the disjoint union of its connected components. We will show that, for every 
ko, every q inSko{D) and every neighborhood U of q there exists ki ^ ko such that 
U n Ski 7^ 0- This implies that the number of connected components of S{D) is 
infinite and that each of the sets Xm = S{D) \ W^^-^^SkiD) is open and dense in 
S{D). Then since S{D) is compact by Proposition 1 and since fl^^^^X^ = 0, the 
Baire Category Theorem gives a contradiction. 

Let Sko{D) be a component of S{D) and q G Sko be such that there exists a 
neighborhood U of q that does not contain points from Sk{D) with k ^ ko. Since 

n^ J i- 1 ; — — ] ;„j-„ c / 7->^ ;„ j- „ — j- rpi r 
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decreasing U if necessary we can assume that Sko{D) \ U ^ Let {/,} C Aut{D) 
be such that /-,■ — > q, — > r in all of D, with r e S{D) as j — > oo. 

Suppose first that r e Sko{D). Then by [B], for any other connected component 
Ski (-^) '^(-^) with ki ^ ko, one has fj {Sk-^ {D)) C U for all sufficiently large j; this 
is impossible since U does not contain an entire component of S{D). If r ^ Sko{D), 
then for all sufficiently large j, fj{Ski^{D)) C U which is again impossible. 

Thus, S{D) either is connected or has uncountably many components. 

The theorem is proved. ■ 



As we noted in the proof of Theorem 2 above, for a smoothly bounded pseudo- 
convex domain of finite type, the set S{D) is invariant under the extension of an 
automorphism of D to the boundary. In the following proposition we show that 
S{D) is generically the smallest invariant subset of dD. 

Proposition 3. Let D C he a smoothly hounded pseudoconvex domain of finite 
type with non-compact automorphism group. Suppose that A C dD is non-empty, 
compact and invariant under Aut{D). Assume further tha t A is not a one-point 
subset ofS{D). Then S{D) C A. 

In particular, if Aut{D) does not have fixed points in dD, then S{D) is the 
smallest compact subset of dD invariant under Aut{D). 

Proof. Since A is closed, it is sufficient to show that every point of S{D) belongs 
to A. Let q e S{D) and {/-,} C Aut(D) be such that fj q, f~^ ^ r in all of D 
as j — > oo, for some r e S{D). Since A is not a one-point subset of S{D), there 
is a point a E A, a ^ r. Then, by [B], fj{a) — > as j — > oo. Since A is invariant 
under any fj, we see 

that fj{a) G A for all j and thus q is either an accumulation point for A, or, if 
fj{a) — q for some index j, then q E A. 

The proposition is proved. ■ 



We now derive several corollaries from the above proposition regarding particular 
sets A. 

Fix < k < n — 1 and denote by L^^D) the set of all points from dD where 
the rank of the Levi form of dD does not exceed k. Clearly, each set Lk{D) is a 
compact subset of dD and is invariant under any automorphism 

of D. Let li denote the minimal rank of the Levi form on dD and I2 the minimal 
rank of the Levi form on dD \ Li^{D). For these sets. Proposition 3 gives the 
following corollary (that was first proved in [H]). No te that the proof in [H] was 
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Corollary 4. Let D be as in Proposition 3. Then either 

(t) S{D)^Li^{D), 
or 

(a) Li^{D) is a one-point subset of S{D) and S{D) C Li^{D). 

Proof. If Li^(D) is not a one-point subset of S{D) then, by Proposition 3, S{D) C 
Li^{D). Suppose now that Li^{D) is a one-point subset of S{D). Then, since 
Li^[D) is strictly contained in Li^{D), one has S{D) C Li^{D). 

The coroUary is proved. ■ 

By a similar argument, one can endeavor to prove an analogous property of the 
type T{q), q e dD, in the sense of D'Angelo. Indeed, denote by Tk{D) the set of all 
points q G dD where r(g) is at least k. We choose ti and t2 such that Tt^{D) ^ 0, 
t2 < ti, and there exists a point of type t2 in dD \ Tt^{D). Since r is invariant 
under automorphisms of -D, so is every set Tk{D). However, the sets Tk{D) do not 
have to be closed, as the type function r may not be upper-semicontinuous on dD 
(see e.g. an example in [D'A2], p. 136). Therefore, for the type we only have a 
somewhat weaker result. 

Corollary 5. Let D be as in Proposition 3. Then either 

(i) S{D) C T^jD), 
or 

(ii) Tt^{D) is a one-point subset of S{D) and S{D) C Tt^{D). 

In place of the type function r, one can consider the multiplicity function /j, 
on dD (see [D'A2], p. 145 for the definition), which is also invariant under the 
extensions of automorphisms to dD. It should be noted that, for q e dD, the 
number T{q) is finite if and only if ii{q) is finite. In contrast with r, however, the 
function /j, is upper-semicintinuous on dD. Analogously to what we have done above 
for the function r, denote by Mk{D) the set of all points q G dD, where n{q) is at 
least k and choose mi and m2 such that nii = maXq^gD IJ'{q), 'n^2 < ^i-, and there 
exists a point of multiplicity m2 in dD \ (D). Due to the upper-semicintinuity 
and invariance of yu, each set Mk{D) is a compact subset of dD that is invariant 
imdcr Aut(D). This observation gives the following analogue of Corollary 6 for 

Corollary 6. Let D be as in Proposition 3. Then either 

(t) S{D)^MmAD), 
or 

(ii) Mmi{D) is a one-point subset of S{D) and S{D) C MmsiD). 

The proof of Corollaries 5, 6 is completely analogous to that of Corollary 4. 
Remarks. 

1. It is plausible that Theorem 2 and Corollaries 4-6 hold without the assump- 
tions of pseudoconvexity and finite type. 

2. We note that, in complex dimension 2, the type r is upper-semicontinuous. 
As a result. Corollary 5 can be stated in this case without passing to the closures of 
the Tf .. Also, in complex dimension 2, Corollary 5 is a consequence of the explicit 
classification of smoothly bounded pseudoconvex domains of finite type with non- 
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3. For a smoothly bounded circular domain, the set S{D) clearly cannot be a one- 
or two-point set. Thus, Theorem 2 gives that, for smoothly bounded pseudoconvex 
circular domains of finite type, S{D) is always a perfect set. Next, since for such 
domains the automorphism group cannnot have fixed points on the boundary , 
Corollary 4 implies that in this case the Levi form of dD has constant rank along 
S{D) and minimizes its rank over dD on S{D) (see also [H]). It also should be noted 
here that, by the results of [BP2], every smoothly bo unded convex domain of finite 
type with non-compact automorphism group is biholomorphically equivalent to a 
certain polynomially defined domain that admits an action of the two-dimensional 
torus T^. Therefore, for any such a domain, S{D) also is a perfect set, and the 
rank of the Levi form is constant and minimal on S{D). 

4. The results of [FIK] imply that, for a smoothly bounded Reinhardt domain D, 
the type is constant along S{D) and maximizes on S{D) the type over dD. It is an 
interesting question whether there exists an analogue of this fact for more general 
domains (cf. Corollaries 5, 6). Note that one can make a statement analogous 
to Corollary 6 for the multitype introduced in [C], since the multitype function is 
upper-semicontinuous with respect to lexicographic ordering. 

5. It also follows from [FIK] that, for a smoothly bounded Reinhardt domain 
D, the real dimension of any orbit of the action of Aut{D) on D is at least 2{k + 
1), where k is the rank of the Levi form of dD along S{D). Moreover, there 
is precisely one orbit of minimal dimension 2{k + 1) (see [K] for a discussion of 
this phenomenon). Also, the orbit of minimal dimension approaches every point of 
S{D) non-tangentially, whereas any other orbit approaches every point of S{D) only 
along tangential directions. It would be interesting to know if similar statements 
hold for more general, e.g. circular, domains. The fact that there exists an orbit 
that approaches S{D) non-tangentially would be very important for a proof of the 
Greene/Krantz conjecture. It also could be used to show that S{D) is a smooth 
submanifold of dD. 

We wish to conclude this paper with a list of immediate open problems that arise 
from the above discussion and which complement some of the preceding remarks. 

Open Problems. 

1. For a smoothly (C°°) bounded domain D C C"^, can the set S{D) be a one- 
or two-point set? Note that the reference [GK3] gives an example of a domain with 
C^~^ boundary, for which S{D) has only two points. It appears that this example 
can be modified, using a parabolic group of automorphisms, so that S{D) has 
just one point. We shall explore this matter further, and additionally investigate 
increasing the boundary smoothness, in a future paper. Indications are that the case 
of finite boundary smoothness will be different from the case of infinite boundary 
smoothness. 

2. For a smoothly bounded domain D, can the set S{D) have uncountably many 
components, for example, can it be a Cantor-type set? 

3. For a smoothly bounded domain D, is the set S{D) always a smooth subman- 
ifold of dD7 Note that the results of [FIK] imply that, for a smoothly bounded 
Reinhardt domain, S{D) is always a smooth submanifold of dD that is difi^eomor- 
phic to a sphere of odd dimension. 

4. Is it always true that the rank of the Levi form is in fact constant and minimal 

„i , ] i-i^^i- i- ;„ „ i- i- ] ™ „i , T^\'? 
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